Solving factored MDPs using non-homogeneous partitions

We present an algorithm for aggregating states in solving large MDPs (represented as factored MDPs) using search by successive re nement in the space of nonhomogeneous partitions. Homogeneity is de ned in terms of stochastic bisimulation and reward equivalence within blocks of a partition. Since homogeneous partitions that de ne equivalent reduced-state-space MDPs can have a large number of blocks, we relax the requirement of homogeneity. The algorithm constructs approximate aggregate MDPs from non-homogeneous partitions, solves the aggregate MDPs exactly, and then uses the resulting value functions as part of a heuristic in re ning the current best non-homogeneous partition. We outline the theory motivating the use of this heuristic and present empirical results. In addition to investigating more exhaustive local search methods we explore the use of techniques derived from research on discretizing continuous state spaces. Finally, we compare the results from our algorithms which search in the space of non-homogeneous partitions with exact and approximate algorithms which represent homogeneous and approximately homogeneous partitions as decision trees or algebraic decision diagrams.